Always-on camera sampling strategies

ABSTRACT

Embodiments of the present invention are directed toward providing intelligent sampling strategies that make efficient use of an always-on camera. To do so, embodiments can utilize sensor information to determine contextual information regarding the mobile device and/or a user of the mobile device. A sampling rate of the always-on camera can then be modulated based on the contextual information.

BACKGROUND

Always-on cameras are cameras that can continuously capture images at agiven sampling rate or that are always available for capturing images,which can be used in various applications on a mobile device, such aslife-logging, gesture identification/recognition, indoor-outdoorinference, and more. As processing and storage capabilities of mobiledevices (e.g., mobile phones, tablets, head- or body-mounted cameras,head-mounted or heads-up displays, etc.) continue to increase, so toodoes the potential functionality of such always-on cameras. However, thepower requirements of always-on cameras can be high relative to othersensors, and many images captured by always-on cameras have little or novalue.

SUMMARY

Embodiments of the present invention are directed toward providingintelligent sampling strategies that make efficient use of an always-oncamera. To do so, embodiments can utilize sensor information todetermine contextual information regarding the mobile device, includingthe state of a user of the mobile device. A sampling rate of thealways-on camera can then be modulated based on the contextualinformation.

An example method of adaptively adjusting sensor sampling of analways-on camera, according to the description, includes causing acamera communicatively coupled with a mobile device to sample at asampling rate, determining a state of a user of the mobile device basedon sensor input, and modulating the sampling rate based on thedetermined state of the user.

An example mobile device, according to the disclosure, includes amemory, and a processing unit coupled with the memory and configured toperform functions including causing a camera communicatively coupledwith the mobile device to sample at a sampling rate, determining a stateof a user of the mobile device based on sensor input, and modulating thesampling rate based on the determined state of the user.

An example apparatus, according to the disclosure, includes means forcausing a camera communicatively coupled with a mobile device to sampleat a sampling rate, means for determining a state of a user of themobile device based on sensor input, and means for modulating thesampling rate based on the determined state of the user.

An example non-transitory computer-readable medium, according to thedisclosure, has instructions embedded thereon for adaptively adjustingsensor sampling of an always-on camera. The instructions includecomputer-executable code for causing a camera communicatively coupledwith a mobile device to sample at a sampling rate, determining a stateof a user of the mobile device based on sensor input, and modulating thesampling rate based on the determined state of the user.

Items and/or techniques described herein may provide one or more of thefollowing capabilities, as well as other capabilities not mentioned.Techniques can provide for increased power efficiency, which can resultin longer battery life of a mobile device in which the always-on camerais disposed. Embodiments can also increase data efficiency by reducingdata that may have little or no use. These and other embodiments, alongwith many advantages and features, are described in more detail inconjunction with the text below and attached figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified illustration of embodiments of different mobiledevices that can utilize an always-on camera according to the techniquesdisclosed herein.

FIG. 2 is a simplified input/output diagram illustrating how embodimentsof the invention can utilize sensor and other information in contextualdeterminations, which can influence how a sampling rate of an always-oncamera may be modulated.

FIG. 3 is an illustration of a portion of a building map that can beused for geo-fencing, place of relevance (POR)-based triggering, andother functions, in some embodiments.

FIG. 4 is a flow chart of a process 400 for adaptively adjusting sensorsampling of an always-on camera of a mobile device, according to oneembodiment.

FIG. 5 illustrates an embodiment of a mobile device.

In the appended figures, similar components and/or features may have thesame reference label. Further, various components of the same type maybe distinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

DETAILED DESCRIPTION

The following description is provided with reference to the drawings,where like reference numerals are used to refer to like elementsthroughout. While various details of one or more techniques aredescribed herein, other techniques are also possible. In some instances,structures and devices are shown in block diagram form in order tofacilitate describing various techniques.

“Instructions” as referred to herein relate to expressions whichrepresent one or more logical operations. For example, instructions maybe “machine-readable” by being interpretable by a machine for executingone or more operations on one or more data objects. However, this ismerely an example of instructions and claimed subject matter is notlimited in this respect. In another example, instructions as referred toherein may relate to encoded commands which are executable by aprocessing unit having a command set which includes the encodedcommands. Such an instruction may be encoded in the form of a machinelanguage understood by the processing unit. Again, these are merelyexamples of an instruction and claimed subject matter is not limited inthis respect.

Always-on cameras in mobile devices are cameras that can provide aconstant stream of images for use without requiring user initiation. Asused herein, the term “always-on camera” does not literally mean thecamera can never be turned off and does not literally mean that thecamera is truly on all the time. Instead, the term “always-on camera” isused to describe a camera which is continuously available over anextended period of time. Such functionality can be provided as abackground service of a mobile device, utilized in any of a variety ofapplications executable by a mobile device, such as life-logging,gesture identification/recognition, indoor-outdoor inference, and more.Although this functionality may be provided at the application layer, itmay additionally or alternatively be provided at a lower level,integrated, for example, into an operating system, device drivers,and/or hardware of a mobile device. By constantly taking images, thealways-on camera of a mobile device can help ensure that there is nearlyalways a recent camera image available for use by an application, ratherthan waiting for a request for a camera image before capturing an imagewith the camera.

One difficulty that can arise when implementing an always-on camera isits high power consumption relative to other components, which canquickly drain battery power of mobile devices. For instance, moderncameras incorporated into mobile devices such as mobile phones andportable gaming systems can consume roughly 70-80 mA per image captured.Even if this relatively high power consumption is reduced, an always-oncamera could still be a high power burden on a mobile device due to thevolume of images it captures. Thus, techniques described herein provideintelligent sampling strategies to increase the efficiency of analways-on camera.

An always-on camera can be included in and/or communicatively coupledwith any of a variety of mobile devices. FIG. 1, for example, is asimplified illustration of embodiments of different mobile devices 100that can utilize an always-on camera. The mobile devices 100 include amobile phone 100-1, a tablet 100-2, and a head-mounted display (HMD)100-3. Although not pictured in FIG. 1, other mobile devices such aspersonal digital assistants (PDAs), notebook computers, personal mediaplayers, gaming devices, cameras, video recorders, and the like, canalso utilize the techniques disclosed herein, as well as one or morewearable devices (e.g., helmet cameras, sports cameras, etc.), which maybe distributed across a user's body. Multiple devices (e.g., wearabledevices) may be communicatively connected with and/or managed by acontextual determination engine, as described in further detail below,in relation to FIG. 2. Furthermore, the style, size, functionality,and/or other features of a mobile device may vary between differentmobile devices of the same type. A person of ordinary skill in the artwill recognize many modifications to the components illustrated.

Mobile devices 100 can include one or more cameras 110 positioned at anyof a variety of locations on the mobile device 100. The mobile phone100-1, for instance, may include a front-facing camera 110-1 and/or arear-facing camera 110-2. The tablet 100-2 can also include afront-facing camera 110-3 and/or additional cameras 110-4 on othersurfaces (e.g., sides and/or back), which may face other directions. TheHMD 100-3 can have an outward-facing camera 110-5 located between thedisplays 120 of the HMD 100-3, as shown, and/or at another location ofthe HMD 100-3.

Any or all of the cameras 110 of a mobile device 100 may be utilized asan always-on camera, depending on desired functionality. Although someembodiments may utilize a particular camera dedicated for always-onfunctionality with particular features that can accommodate always-onfunctionality (e.g., a wide-angle, fisheye, low-power, low resolution,etc.), embodiments may additionally or alternatively utilize any of avariety of camera types (e.g., infrared, ultraviolet, spectrometer, highresolution, front-facing, etc.) for always-on functionality. Embodimentsin which a mobile device 100 includes a plurality of cameras may choosewhich camera to use as an always-on camera and/or may toggle betweendifferent cameras in certain situations. Additional details regardingthis functionality are provided below.

Embodiments of a mobile device 100 may vary substantially from themobile devices shown in FIG. 1. In addition or as an alternative tocameras embedded within a mobile device 100, embodiments may include amobile device physically separate from a camera, but communicativelycoupled therewith (e.g., via wireless or wired technologies). Othersensors may also be physically separate from with a mobile device 100,yet in communication therewith, such that cameras and/or other sensorsmay be distributed, for example, at various locations on and/or aroundthe body of a user.

FIG. 2 is a simplified input/output diagram illustrating how embodimentsdescribed herein can utilize sensor and other information in contextualdeterminations, which can influence how a sampling rate of an always-oncamera may be modulated. These contextual determinations can be made bya contextual determination engine 280 executed, for example, by softwareand/or hardware of the mobile device 100 (as described in further detailbelow with regard to FIG. 5) and/or may be part of and/or integratedinto a larger software and/or hardware application of the mobile device100. The contextual determination engine 280 can receive data fromvarious components in order to make the contextual determination,including light sensor(s) 210 (which can include, for example, ambientlight sensor(s), Ultra violet (UV) sensor(s), UV-A sensor(s), UV-Bsensor(S), red-green-blue (RGB) sensor(s), and the like), microphone(s)220, motion/orientation detector(s) 230 (which can include, for example,one or more gyroscopes, magnetometers (and/or another types ofcompasses), accelerometer(s), and the like), camera(s) 110, wirelesscommunication interface 240, (which can include, for example, 2G, 3G, 4Gmodems, WiFi, Bluetooth, Bluetooth LE, WiMax, and the like) satellitepositioning sensor(s) 250, and/or other sensor(s) 260 (e.g.,altimeter(s), proximity sensor(s), compressive imaging sensor(s),temperature sensor(s), pressure sensor(s), touch sensor(s), fingerprintsensor(s), and the like). Components may also include map(s) 270, motionmodel(s) 290, and/or application data 295. A person of ordinary skill inthe art will recognize many modifications to the components illustratedin FIG. 2. Components can be, for example, added, omitted, combined,and/or separated. Depending on desired functionality, components may beintegrated into the mobile device 100 and/or separate from the mobiledevice 100 (communicating, for example, via a wired and/or wirelessconnection, as previously described).

The one or more cameras 110 can be integrated into and/orcommunicatively coupled with a mobile device 100. Furthermore, any orall of the camera(s) 110 can be designated as an always-on camera. Thecontextual determination engine 280 can make this designation. Not onlymay the sample rate of an always-on camera be impacted by a contextualdetermination of the contextual determination engine 280, but imagesfrom an always-on camera (and/or other camera(s) 110) can be processedfor further contextual determinations.

Techniques herein provide for contextual determinations that can resultin modulating a sampling rate of an always-on camera (which can be aninitial sampling rate chosen by a software application and/or hardwaredefault). The sampling rate can be modulated to any of a wide variety ofsampling rates. For example an initial sampling rate may be one sampleevery 4-5 seconds. However, certain contextual determinations may causethe sampling rate to be increased to 30 samples per second (i.e., 30frames per second (FPS)) or more. Other determinations may result inreducing the sampling rate to, for example, once every 5 minutes and/orsuppressing samples altogether. Ultimately contextual determinations mayresult in sampling rates from zero (i.e., no sampling) to the highestrate feasible under the hardware and/or software constraints of thealways-on camera. Additionally or alternatively, as described in moredetail below, embodiments may provide for selectively triggering analways-on camera and/or selectively activating certain cameras in amulti-camera system based on contextual determinations.

Light sensor(s) 210 can include any of a variety of photo-sensitivesensors such as active light sensor(s), RGB sensor(s), Ultra violet (UV)sensor(s) and the like. Such light sensor(s) 210 typically consume(s)far less power than an always-on camera, and can be useful indetermining the context of a mobile device 100 and/or user of the mobiledevice. For example, one or more light sensors 210 can be positioned todetermine the light intensity to which an always-one camera is exposed.If detected light intensity is below a certain threshold, the contextualdetermination engine 280 may determine that an always-on camera is in apocket or purse or in a darkened room, in which case sampling of thealways-on camera can be reduced or suspended.

Some embodiments may have multiple cameras 110 that may be utilized asan always-on camera. Where lighting conditions could vary by cameralocation, and one or more light sensors 210 are positioned to correspondwith each camera, the contextual determination engine 180 can determinewhich of the multiple cameras 110 to utilize as an always-on camerabased on light intensity data from the light sensor(s) 210.

Additionally or alternatively, the contextual determination engine 180may use motion/orientation detector(s) 230, motion model(s) 290, and/orlight sensor(s) 210 to determine the position of a mobile device and/oralways-on camera position relative to a user (such as on a desk or inthe user's hand, bag, shirt pocket, pant pocket, holster, etc.), andmodulate a sampling rate of the always-on camera accordingly. Forexample, if the mobile device is detected in a pocket, bag, purse, orholster and/or is exposed to light below a certain threshold, the viewto the always-on camera is likely obstructed, and the sampling rate ofthe always-on camera can be reduced or stopped. In a scenario in whichcaptured images of an always-on camera are used by a life-loggingapplication used to automatically collect a photo log of a user's life,this functionality could result in avoiding unnecessary image captureand significant power savings when the photos provide little or nouseful information, such as during night time, when the always-on camerais pocket, bag, and so forth. Motion/orientation detector(s) 230 and/ormotion model(s) 290 may also enable the contextual determination engine280 to determine a state of a user who is carrying and/or associatedwith the mobile device 100. The determinable user states may vary basedon desired functionality. Examples include walking, running, exercising,in transit, and other such activities that may involve detectablemovement and/or orientation. Furthermore, the user state can impact howan always-on camera is modulated. For example, an always-on camera maybe configured to increase a sampling rate where the user state isindicative of movement outdoors (e.g., where a user's environment islikely to frequently change). On the other hand, a user state indicativeof a relatively inactive state indoors may cause the contextualdetermination engine 280 to reduce the sampling frequency of analways-on camera.

A contextual determination engine 280 can also utilize data from themicrophone (s) 220 to detect certain audio activity and direction. Thiscan be used to determine a particular activity is going on (e.g., ameeting or conversation). This can be used to modulate a samplingfrequency of an always-on camera, increasing the sampling, for example,when sound is detected.

Embodiments in which a mobile device can utilize any of multiple cameras110 as an always-on camera, audio directionality derived from microphonedata can be used to determine which direction a sound is coming from. Acorresponding camera likely facing the direction of the sound may thenbe designated as the always-on camera to capture images based on thedirection of current audio activity, which is also likely to holdinformative visual information such as a speaker identity, emotions,expressions, visual scene corresponding to audio activity, and the like.The microphone(s) 220 can include one or more low-power microphoneactivity detectors to determine audio directionality.

This functionality can be used, for example, in a scenario in which amobile device 100 with multiple cameras 110 is executing a life-loggingapplication. In a meeting context, a user can place the mobile device100 on a table, at which point the contextual determination engine 280may use microphone(s) 220 and/or other components to determine a userstate (e.g., “in a meeting”), and leverage audio directionalityinformation of the microphones(s) 220 to determine which camera 110 tosample from and/or how to modulate (e.g., increase or decrease) acamera's sampling rate, based on the user state. When a user is workingin his or her office, the contextual determination engine can utilizeaudio data (e.g., using audio environment clustering and/or other audioanalysis techniques) to determine this user state (e.g., “working inoffice”), and reduce the sampling rate or turn off an always-on camera,based on the user state, until a change in audio ambience is observed.

The contextual determination engine 280 can also use data frommotion/orientation detector(s) 230 to determine contextual information,such as an activity in which the user may be engaged (e.g., walking,running, in transit, etc.). Data from the motion/orientation detector(s)230 can be compared against motion model(s) 290, which can modeldifferent activities and/or other states of a user, to make thedetermination. If, for example, sensed motion is within a thresholdlikeness of modeled motion of a certain activity, the contextualdetermination engine 280 may determine that the user is engaged in thatactivity. In some embodiments, data can be processed by a classifiersuch as an absolute motion classifier or a relative or full motionclassifier, or may be processed by a pedometer module or function.

The contextual determination engine 280 can modulate the sampling rateof an always-on camera based on a speed, motion, and/or determinedcontext of a user (e.g., a user state) and/or mobile device 100. Forexample, if the contextual determination engine 280 determines that auser is engaged in certain activities that involve a relatively highamount of movement, such as running or jumping. Thus, the contextualdetermination engine 280 can reduce or suspend sampling from thealways-on camera.

Similarly, the contextual determination engine 280 can determine that auser is traveling at a certain speed by estimating the speed frommotion/orientation detector(s) 230 and/or location information (whichcan be determined using, for example, the satellite positioningreceiver(s) 250, a wireless communication interface 240 capable ofproviding positioning and/or speed information, and the like). If thespeed is above a certain threshold, the contextual determination engine280 can reduce or suspend sampling from the always-on camera to reducethe likelihood of capturing blurred images. Additionally oralternatively, speed may be used in a determination of a user state,which may then be used to determine how sampling of the always-on cameracan be modulated.

The contextual determination engine 280 can also use orientationinformation from the motion/orientation detector(s) 230 to determinewhether to modulate the sampling rate of an always-on camera and/ordesignate a camera for always-on functionality based on sensedorientation and/or motion when combined with one or more othercomponents. For example, the contextual determination engine 280 maysuppress image capture of an always-on camera if the previous imagetaken by the always-on camera is determined to be uninformative for acertain use-case (e.g., pointing towards the sky or ceiling), and themobile device 100 and/or always-on camera has been at absolute restsince then.

Geo-fencing and place of relevance (POR)-based triggers can also be usedin contextual determinations that may impact the sampling rate of analways-on camera. FIG. 3 is an illustration of a portion of a buildingmap 300 that can be used for geo-fencing and/or POR-based triggering. Insome embodiments, the building map 300 may be provided by a buildingserver for indoor positioning and/or other purposes. Optionally, amobile device 100 may download a map from a data communication network,such as the internet. The building map 300 and/or other maps may beincluded in the map(s) 270 illustrated in FIG. 2, which may be at leastpartially stored in a memory of the mobile device. It will be understoodthat the building map 300 is provided as a non-limiting example.Geo-fencing and/or POR functionality described herein can utilize any ofa variety of map types besides building maps.

For an embodiment utilizing a building map for geo-fencing, sensor datafrom satellite positioning sensor(s) 260, motion/orientation detector(s)230, and/or other sensor data may be used, in conjunction the buildingmap 300, by the contextual determination engine 280 to determine that auser is at a certain location, which may impact whether to increase ordecrease the sampling rate of an always-on camera. Certain locations ofthe map 300 may be “tagged” (i.e., associated with) informationindicative of a user activity or other contextual information, which canimpact the frequency at which an always-on camera is sampled. Forexample, a restroom 320 and/or conference room 310 may be tagged assuch, enabling the contextual determination engine 280 to infer certainuser activities when a mobile device is determined to be located ineither of these locations. Other such special-purpose locations can alsobe tagged to imply certain user activity (e.g., cafeteria, theater,garage, bedroom, elevator, etc.).

Some locations may clearly require a relatively high or relatively lowsampling rate for an always-on camera. For example, the sampling ratemay be reduced/suspended in a bathroom, bedroom, high-security area,and/or other location in which image capture may be prohibited ordiscouraged. On the other hand, the sampling rate of an always-on cameramay be increased for certain PORs, such as theme parks and/or otherlocations in which a user and/or application may want a relatively highrate of image capture. Depending on desired functionality, a user may beable to define the boundaries of the PORs and/or tag certain PORs withinformation that may impact a chosen sampling rate. In some embodiments,the contextual determination engine 280 may maintain and/or accessprivacy settings that may inhibit or prevent an always on camera fromtaking pictures. In some embodiments, these settings may be created bythe user. Embodiments may also cause a mobile device 100 to provide analert (e.g., using sound, vibration, an indication on a display, etc.)when a user enters a private or secure area and the always-on camerawill suspend image capture. A similar alert may additionally oralternatively be provided when a user exits the private or secure area,and the always-on camera will resume image capture.

Image data and/or other information from camera(s) 110, including imagesby an always-on camera, can be utilized by the contextual determinationengine 280 to duty cycle and/or time stagger image capture of analways-on camera. Image processing techniques can be utilized on one ormore images from the camera(s) 110 to detect movement, which mayindicate that there is a change in the mobile device's surroundings orthat a certain activity is taking place. For instance, movement can bedetected from changes in successively-captured images. If, for example,the camera is facing a scene in which no movement is detected for thelast two images captured, the contextual determination engine 280 maydetermine that the context of the mobile device 100 is static or thestate of a user includes an activity in which very little change in theuser's environment occurs, in which case the contextual determinationengine 280 may implement an exponential back-off type duty cycling toreduce power consumption.

Additionally or alternatively, where a mobile device has multiplecameras, it may cycle through the cameras (staggering captures fromeach) or synchronize captures from all cameras. For example, whereimages from an always-on camera indicate that little or no movement isoccurring within the camera's view, the contextual determination engine280 can designate another of the cameras 110 as the always-on camera,rotating from one camera 110 of the mobile device 100 to the next untilmovement (and/or some other trigger) is detected. In some embodiments, acontextual determination engine may optionally synchronize the camera(s)110 to capture images at the same or approximately the same time.

The contextual determination engine 280 may further utilize a wirelesscommunication interface 240 to make a contextual determination that mayimpact the sampling rate of an always-on camera. The wirelesscommunication interface 240 can detect the presence of wireless signals,such as near-field communication (NFC) signals, IEEE 802.11 (e.g.,WiFi), Bluetooth (BT), Bluetooth Low Energy (BT-LE), and the like. Achange in wireless signals can be indicative of a change in the mobiledevice's surroundings and/or location, including the presence and/orloss of a wireless device. By means of an example, a change in thenumber of Bluetooth devices in the vicinity of the device might be anindication that new devices and/or people have entered or left. Also,the frequency of the change in the number of devices could indicate thedynamism of the environment (e.g., a shopping mall may have more changecompared to home). This change in wireless signals can be used as atrigger to modulate the sampling rate of an always-on camera. That is,such wireless triggers can denote that the mobile user's immediatevicinity has changed which may be cause to increase the sampling rate ofan always-on camera. The wireless communication interface 240 mayadditionally or alternatively utilize a radio access technology (RAT)switch, for example, when a mobile device 100 switches from using onewireless technology to another, to determine a change in wirelesssignals.

The contextual determination engine 280 may further utilize applicationdata 295 obtained from one or more applications executed by the mobiledevice 100, which can be indicative of the state of a user. Such datacan include, without limitation, calendar information, locationinformation, social media status updates, and more. For example,information from a calendar application may indicate that the user is in(or likely in) a meeting. Based on that determined user state, thecontextual determination engine may choose to modulate a sampling rateof the camera accordingly to help ensure a proper amount of visualinformation is captured. Application data can vary substantially, basedon application functionality.

Referring again to FIG. 3, in addition providing geo-fencing and/orPOR-related information, a building map 300 (or other map type, such asa room, courtyard, etc.) can be used to determine whether what may be inthe view of the always-on camera (e.g., looking at a wall, a ceiling,out a window, a stage, etc.), which can impact its sampling rate. Forexample, using the building map 300 along with orientation and/orlocation information, the contextual determination engine 280 maydetermine that the mobile device 100 is in a room 310 with the always-oncamera facing a wall 313. Because the wall is likely contains little orno useful data, the contextual determination engine 280 may reduce inthe always-on camera's sampling rate. On the other hand, if the room 310is a conference room and the always-on camera is determined to be facinga projector screen 315, then the sampling rate may be increased.Additionally or alternatively, where multiple cameras are available,this information can be used to determine which camera(s) 110 to sample(e.g., designate as an always-on camera), sampling from the camera(s)110 with views that are more likely to include items of interest.

The techniques and/or sensors described above can be prioritized and/orutilized in conjunction with one another, depending on the situation.For example, a microphone may be utilized with building map data todetermine which camera(s) 110 to designate as always-on cameras. Inanother example, the contextual information determined from the lightsensor(s) 210 may be prioritized over geo-fencing information such that,if the light sensor(s) 210 indicate the always-on camera (and/or mobiledevice 100) is in a user's pocket, the always-on camera's sampling ratewill be reduced, even if the mobile device 100 is determined to be in aconference room, which would have increased the sampling rate otherwise.

FIG. 4 is a flow chart of a process 400 for adaptively adjusting sensorsampling of an always-on camera of a mobile device, according to oneembodiment. The process 400 can be executed by various hardware and/orsoftware components of a mobile device 100, such as a contextualdetermination engine 280, sensors, and/or other components illustratedin FIG. 2. In particular, means for performing the process 400 caninclude, for example, specialized and/or generalized hardware can beprogrammed and/or otherwise configured to perform all or part of theprocess 400 shown. Such means are described in further detail below withregard to FIG. 5.

The process 400 can optionally begin at block 410 by selecting one ormore camera(s) from a plurality of cameras, based on contextualinformation. Such information can include, for example, maps (which canbe indicative of the camera's view relative to certain mapped features),orientation and/or motion information, sound directionality, featuresfrom one or more previously-captured images, and the like. Certaincameras, such as a wide-angle or fish-eye camera, can be used as adefault camera for always-on functionality if little or no contextualinformation is available, or if the context is not easily determinable.Means for performing the function of block 410 can include, for example,one or more processing units, sensors, and/or software (e.g., anapplication, operating system, etc.), which are described in furtherdetail below in FIG. 5.

At block 420, the process 400 includes causing a camera of a mobiledevice to sample at a sampling rate. As described above, such always-onfunctionality can be a background service that is incorporated into anapplication executed by the mobile device, an operating system of themobile device, a device driver, and/or physical layer of the mobiledevice to provide automatic image capture without user input. An initialsampling rate may be determined by user and/or application settings,camera defaults, mobile device defaults, and the like. Means forperforming the function of block 410 can include, for example, one ormore processing units, cameras, a wireless communication interface,and/or software, which are described in further detail below in FIG. 5.

At block 430 the state of a user of the mobile device is determined,based on sensor input. As described previously, contextual information,such as the state of a user, can be determined using input from any of avariety of sensors and/or information from other sources. As discussedabove, data can be received from a camera, a motion sensor, a lightsensor, an orientation sensor, a microphone, a wireless communicationinterface, and/or a satellite receiver.

Determining the state of a user can include determining any of a varietyof contextual information about a user. This can include, for example, aposition of the mobile device in relation to a user of the mobile device(e.g., in a pocket, in a purse, in a holster, in the user's hand, on adesk near the user, etc.), an activity engaged in by a user of themobile device (e.g., in a meeting, on a telephone call, being alone,watching TV, playing, reading, writing, meeting people, looking atscreen, giving a presentation, eating, shopping, cooking, attending atalk, exercising, listening to music, sleeping, etc.) a motion state ofthe user (e.g., stationary, walking, running, in transit, fiddling withthe mobile device, etc.), a POR (place of relevance) of the mobiledevice (e.g., at a location that can be indicative of a state of theuser, such as in a cafeteria, at a bus stop, in a conference room,etc.), a change in the mobile device's surroundings (e.g., an audiochange, change in ambient light information, change in the number ofdevices around the user, change in the number of people around the user,etc.), and the like. Determining the state of a user additionally oralternatively may include classifying the user's environment as beingone or more of a variety of predetermined states, such as in a meeting,on a telephone call, being alone, and the like. This type ofenvironmental classification can come from information about a POR, astate of a mobile device (e.g., on a telephone call), sensor data,and/or other information. Determining a state of the user mayadditionally include clustering sensor input from one or more sensors.For example, if a user is working along in the office, audio environmentclustering techniques can be used to determine this state of the user.Corresponding image capture by an always-on camera can be minimized orturned off until a change in audio ambience is observed. Means forperforming the function of block 410 can include, for example, one ormore processing units, cameras, a wireless communication interface,memory (which can be used, for example, to store motion models, maps,and/or other data), and/or software, which are described in furtherdetail below in FIG. 5.

At block 440, the sampling rate is modulated based on the determinedstate of the user. Modulating an existing sampling rate can includeincreasing or decreasing a default or initial sampling rate. In someinstances, modulating the sampling rate may also include suppressing oneor more samples that would otherwise have been sampled pursuant to thedefault or initial rate. As indicated previously, the sampling rate maybe modulated to virtually any rate the always-on camera is capable of,based on the determined state of the user. This can include, forexample, suspending sampling of the always-on camera altogether,increasing image capture to video rates (e.g., 30 images per second) orhigher, or selecting a sampling rate anywhere in between. Means forperforming the function of block 410 can include, for example, one ormore processing units, cameras, a wireless communication interface,and/or software, which are described in further detail below in FIG. 5.It can further be noted that, not only can the determined state of theuser influence how a sampling rate of an always-on camera is modulated,but images from the always-on camera can be used to determine a userstate. Thus, determining user state can occur both before and afterimage capture by the always-on camera.

It should be appreciated that the specific steps illustrated in FIG. 4provide an example process 400 for adaptively adjusting sensor samplingof an always-on camera of a mobile device. Alternative embodiments mayinclude alterations to the embodiments shown. Furthermore, additionalfeatures may be added or removed depending on the particularapplications. For example, a second camera may be selected to be thealways-on camera after determining the state of the user. In amulti-camera system, sampling may be rotated and/or synchronized betweencameras, as discussed previously herein. One of ordinary skill in theart would recognize many variations, modifications, and alternatives.

FIG. 5 illustrates an embodiment of a mobile device 100, which caninclude and/or be communicatively coupled with an always-on camera, asdescribed above. The mobile device 100 also can be configured to performthe methods provided by various other embodiments, such as the process400 illustrated in FIG. 4. It should be noted that FIG. 5 is meant onlyto provide a generalized illustration of various components, any or allof which may be utilized as appropriate. Furthermore, the hardwareand/or software components of the mobile device 100 shown in FIG. 5 canbe configured to implement one or more of the components illustrated inFIG. 2, such as the contextual determination engine 280, sensors, andmore.

It can also be noted that components illustrated by FIG. 5 can belocalized to a single mobile device and/or distributed among variousnetworked devices, which may be disposed at different physicallocations. For example, some embodiments may include distributedcamera(s) 546 and/or other sensors 540 at various locations on or near auser's body. An always-on camera of an HMD (worn on a user's head), forinstance, may be communicatively coupled with a mobile phone in theuser's pocket, and components shown in FIG. 5 may be distributed amongthe HMD and mobile phone in any of a variety of ways, depending ondesired functionality.

The mobile device 100 is shown comprising hardware elements that can beelectrically coupled via a bus 505 (or may otherwise be incommunication, as appropriate). The hardware elements may includeprocessing unit(s) 510 which can include without limitation one or moregeneral-purpose processors, one or more special-purpose processors (suchas digital signal processors (DSPs), graphics acceleration processors,application specific integrated circuits (ASICs), and/or the like),and/or other processing structure or means, which can be configured toperform one or more of the methods described herein, including method ofFIG. 4. The mobile device 100 also can include one or more input devices570, which can include without limitation a touch screen, a touch pad,button(s), dial(s), switch(es), and/or the like; and one or more outputdevices 515, which can include without limitation a display, lightemitting diode (LED), speakers, and/or the like.

The mobile device 100 might also include a wireless communicationinterface 530, which can include without limitation a modem, a networkcard, an infrared communication device, a wireless communication device,and/or a chipset (such as a Bluetooth™ device, an IEEE 802.11 device, anIEEE 802.15.4 device, a WiFi device, a WiMax device, cellularcommunication facilities, etc.), and/or the like. Communication to andfrom the mobile device 100 may thus also be implemented, in someembodiments, using various wireless communication networks. Thesenetworks can include, for example, a wide area wireless network (WWAN),a wireless local area network (WLAN), a wireless personal area network(WPAN), and the like. A WWAN may be a Code Division Multiple Access(CDMA) network, a Time Division Multiple Access (TDMA) network, aFrequency Division Multiple Access (FDMA) network, an OrthogonalFrequency Division Multiple Access (OFDMA) network, a Single-CarrierFrequency Division Multiple Access (SC-FDMA) network, a WiMax (IEEE802.16), and so on. A CDMA network may implement one or more radioaccess technologies (RATs) such as cdma2000, Wideband-CDMA (W-CDMA), andso on. Cdma2000 includes IS-95, IS-2000, and/or IS-856 standards. A TDMAnetwork may implement Global System for Mobile Communications (GSM),Digital Advanced Mobile Phone System (D-AMPS), or some other RAT. AnOFDMA network may implement Long Term Evolution (LTE), LTE Advanced, andso on. LTE, LTE Advanced, GSM, and W-CDMA are described in documentsfrom a consortium named “3rd Generation Partnership Project” (3GPP).Cdma2000 is described in documents from a consortium named “3rdGeneration Partnership Project 2” (3GPP2). 3GPP and 3GPP2 documents arepublicly available. A WLAN may also be an IEEE 802.11x network, and aWPAN may be a Bluetooth network, an IEEE 802.15x, or some other type ofnetwork. The techniques described herein may also be used for anycombination of WWAN, WLAN and/or WPAN. The wireless communicationinterface 530 may permit data to be exchanged directly with othersensors, systems, and/or any other electronic devices described herein.The communication can be carried out via one or more wirelesscommunication antenna(s) 532 that send and/or receive wireless signals534.

The wireless communication interface 530 may also be utilized todetermine a location of the mobile device 100. For example, accesspoints (including base stations and/or other systems used for wirelessvoice and/or data communication) can serve as independent sources ofposition data, e.g., through implementation of trilateration-basedprocedures based, for example, on RTT and/or RSSI measurements. Theaccess points can be part of a WLAN that operates in a building toperform communications over smaller geographic regions than a WWAN.Moreover, the access points can be part of a WiFi network (802.11x),cellular piconets and/or femtocells, Bluetooth network, and the like.The access points can also form part of a Qualcomm indoor positioningsystem (QUIPSTM).

The mobile device 100 can further include sensor(s) 540. As indicatedherein, sensor(s) 540, which can correspond to the sensors described inFIG. 2, can include sensors from which an orientation and/or motion ofthe mobile device 100 can be determined, such as one or moreaccelerometer(s) 541, gyroscope(s) 542, magnetometer(s) 544, and thelike. The mobile device 100 may further include other sensor(s) 540,such as microphone(s) 565, light sensor(s) 546, proximity sensors, andmore, as described previously in relation to FIG. 2. Camera(s) 543 caninclude any number of different cameras with different features (RGB,infrared, wide-angle, fisheye high-resolution, etc.), any or all ofwhich can be utilized as an always-on camera as described herein.

Embodiments of the mobile device may also include a satellitepositioning system (SPS) receiver 580 capable of receiving signals 584from one or more SPS using an SPS antenna 582. The SPS receiver 580 cancorrespond to the satellite positioning receiver(s) 250 described inrelation to FIG. 2, which can provide location information (e.g.,coordinates) regarding the mobile device, as well as information derivedtherefrom (speed, acceleration, etc.). Transmitted satellite signals 584may include, for example, signals marked with a repeating pseudo-randomnoise (PN) code of a set number of chips and may be located on groundbased control stations, user equipment and/or space vehicles. Satellitepositioning systems may include such systems as the Global PositioningSystem (GPS), Galileo, Glonass, Compass, Quasi-Zenith Satellite System(QZSS) over Japan, Indian Regional Navigational Satellite System (IRNSS)over India, Beidou over China, etc., and/or various augmentation systems(e.g., an Satellite Based Augmentation System (SBAS)) that may beassociated with or otherwise enabled for use with one or more globaland/or regional navigation satellite systems. By way of example but notlimitation, an SBAS may include an augmentation system(s) that providesintegrity information, differential corrections, etc., such as, e.g.,Wide Area Augmentation System (WAAS), European Geostationary NavigationOverlay Service (EGNOS), Multi-functional Satellite Augmentation System(MSAS), GPS Aided Geo Augmented Navigation or GPS and Geo AugmentedNavigation system (GAGAN), and/or the like.

The mobile device 100 may further include (and/or be in communicationwith) a memory 560. The memory 560 can include, without limitation,local and/or network accessible storage, a disk drive, a drive array, anoptical storage device, a solid-state storage device, such as a randomaccess memory (“RAM”), and/or a read-only memory (“ROM”), which can beprogrammable, flash-updateable, and/or the like. Such storage devicesmay be configured to implement any appropriate data stores, includingwithout limitation, various file systems, database structures, and/orthe like.

The memory 560 of the mobile device 100 also can comprise softwareelements (not shown), including an operating system, device drivers,executable libraries, and/or other code, such as one or more applicationprograms, which may comprise computer programs provided by variousembodiments, and/or may be designed to implement methods, and/orconfigure systems, provided by other embodiments, as described herein.Merely by way of example, one or more procedures described with respectto the method(s) discussed above, such as the process described inrelation to FIG. 4, might be implemented as code and/or instructionsexecutable by the mobile device 100 (and/or a processing unit within amobile device 100) (and/or another device of a positioning system). Inan aspect, then, such code and/or instructions can be used to configureand/or adapt a general purpose computer (or other device) to perform oneor more operations in accordance with the described methods.

It will be apparent to those skilled in the art that substantialvariations may be made in accordance with specific requirements. Forexample, customized hardware might also be used, and/or particularelements might be implemented in hardware, software (including portablesoftware, such as applets, etc.), or both. Further, connection to othercomputing devices such as network input/output devices may be employed.

As mentioned above, in one aspect, some embodiments may employ acomputer system (such as the mobile device 100) to perform methods inaccordance with various embodiments of the invention. According to a setof embodiments, some or all of the procedures of such methods areperformed by the mobile device 100 in response to processing unit(s) 510executing one or more sequences of one or more instructions (which mightbe incorporated into an operating system and/or other code) contained inthe memory 560. Merely by way of example, execution of the sequences ofinstructions contained in the memory 560 might cause the processingunit(s) 510 to perform one or more procedures of the methods describedherein. Additionally or alternatively, portions of the methods describedherein may be executed through specialized hardware.

The methods, systems, and devices discussed herein are examples. Variousembodiments may omit, substitute, or add various procedures orcomponents as appropriate. For instance, features described with respectto certain embodiments may be combined in various other embodiments.Different aspects and elements of the embodiments may be combined in asimilar manner. The various components of the figures provided hereincan be embodied in hardware and/or software. Also, technology evolvesand, thus, many of the elements are examples that do not limit the scopeof the disclosure to those specific examples.

Specific details are given in the description to provide a thoroughunderstanding of the embodiments. However, embodiments may be practicedwithout these specific details. For example, well-known circuits,processes, algorithms, structures, and techniques have been shownwithout unnecessary detail in order to avoid obscuring the embodiments.This description provides example embodiments only, and is not intendedto limit the scope, applicability, or configuration of the invention.Rather, the preceding description of the embodiments will provide thoseskilled in the art with an enabling description for implementingembodiments of the invention. Various changes may be made in thefunction and arrangement of elements without departing from the spiritand scope of the invention.

It has proven convenient at times, principally for reasons of commonusage, to refer to such signals as bits, information, values, elements,symbols, characters, variables, terms, numbers, numerals, or the like.It should be understood, however, that all of these or similar terms areto be associated with appropriate physical quantities and are merelyconvenient labels. Unless specifically stated otherwise, as is apparentfrom the discussion above, it is appreciated that throughout thisSpecification discussions utilizing terms such as “processing,”“computing,” “calculating,” “determining,” “ascertaining,”“identifying,” “associating,” “measuring,” “performing,” or the likerefer to actions or processes of a specific apparatus, such as a specialpurpose computer or a similar special purpose electronic computingdevice. In the context of this Specification, therefore, a specialpurpose computer or a similar special purpose electronic computingdevice is capable of manipulating or transforming signals, typicallyrepresented as physical electronic, electrical, or magnetic quantitieswithin memories, registers, or other information storage devices,transmission devices, or display devices of the special purpose computeror similar special purpose electronic computing device.

Terms, “and” and “or” as used herein, may include a variety of meaningsthat also is expected to depend at least in part upon the context inwhich such terms are used. Typically, “or” if used to associate a list,such as A, B, or C, is intended to mean A, B, and C, here used in theinclusive sense, as well as A, B, or C, here used in the exclusivesense. In addition, the term “one or more” as used herein may be used todescribe any feature, structure, or characteristic in the singular ormay be used to describe some combination of features, structures, orcharacteristics. However, it should be noted that this is merely anillustrative example and claimed subject matter is not limited to thisexample. Furthermore, the term “at least one of” if used to associate alist, such as A, B, or C, can be interpreted to mean any combination ofA, B, and/or C, such as A, AB, AA, AAB, AABBCCC, etc.

Having described several embodiments, various modifications, alternativeconstructions, and equivalents may be used without departing from thespirit of the disclosure. For example, the above elements may merely bea component of a larger system, wherein other rules may take precedenceover or otherwise modify the application of the invention. Also, anumber of steps may be undertaken before, during, or after the aboveelements are considered. Accordingly, the above description does notlimit the scope of the disclosure.

What is claimed is:
 1. A method of adaptively adjusting sensor samplingof an always-on camera, the method comprising: causing a cameracommunicatively coupled with a mobile device to sample at a samplingrate; determining a state of a user of the mobile device based on sensorinput; and modulating the sampling rate based on the determined state ofthe user.
 2. The method of claim 1, further comprising: selecting thecamera from a plurality of cameras, based on contextual information;wherein causing the camera to sample at the sampling rate is based onthe selection of the camera.
 3. The method of claim 2, furthercomprising determining the contextual information, the determiningcomprising determining at least one of: a location or coordinate of themobile device, a position of the mobile device in relation to the userof the mobile device, an activity engaged in by the user of the mobiledevice, a place of relevance of the mobile device, or a change insurroundings of the mobile device.
 4. The method of claim 1, wherein thestate of the user comprises a motion state of the user.
 5. The method ofclaim 4, wherein the motion state comprises at least one of: stationary,walking, running, in transit, or fiddling with the mobile device.
 6. Themethod of claim 1, wherein determining the state of the user comprisesdetermining the user is involved in at least one activity selected fromthe following list of activities: in a meeting, on a telephone call,being alone, watching TV, playing, reading, writing, meeting people,looking at screen, giving a presentation, eating, shopping, cooking,attending a talk, exercising, listening to music, or sleeping.
 7. Themethod of claim 1, wherein: the sampling rate comprises a default rate;and the modulating comprises suppressing one or more samples that wouldotherwise have been sampled pursuant to the default rate.
 8. The methodof claim 1, wherein the sensor input comprises data from at least oneof: the camera, a motion sensor, a light sensor, an orientation sensor,a microphone, a wireless communication interface, or a satellitereceiver.
 9. The method of claim 8, wherein the determining comprisesclustering sensor input from one or more sensors.
 10. The method ofclaim 1, further comprising determining the state of the user based on amap of a building.
 11. The method of claim 1, further comprisingdetermining the state of the user based on information received from acalendar application executed by the mobile device.
 12. A mobile devicecomprising: a memory; and a processing unit coupled with the memory andconfigured to perform functions including: causing a cameracommunicatively coupled with the mobile device to sample at a samplingrate; determining a state of a user of the mobile device based on sensorinput; and modulating the sampling rate based on the determined state ofthe user.
 13. The mobile device of claim 12, wherein the processing unitis further configured to: select the camera from a plurality of cameras,based on contextual information; wherein causing the camera to sample atthe sampling rate is based on the selection of the camera.
 14. Themobile device of claim 13, wherein the processing unit is configured todetermine the contextual information by determining at least one of: alocation or coordinate of the mobile device, a position of the mobiledevice in relation to the user of the mobile device, an activity engagedin by the user of the mobile device, a place of relevance of the mobiledevice, or a change in surroundings of the mobile device.
 15. The mobiledevice of claim 12, wherein the processing unit is configured todetermine the state of the user by determining a motion state of theuser.
 16. The mobile device of claim 15, wherein the processing unit isconfigured to determine the motion state as at least one of: stationary,walking, running, in transit, or fiddling with the mobile device. 17.The mobile device of claim 12, wherein the processing unit is configuredto determine the state of the user by determining the user is involvedin at least one activity selected from the following list of activities:in a meeting, on a telephone call, being alone, watching TV, playing,reading, writing, meeting people, looking at screen, giving apresentation, eating, shopping, cooking, attending a talk, exercising,listening to music, or sleeping.
 18. The mobile device of claim 12,wherein: the sampling rate comprises a default rate; and the processingunit is configured to modulate the sampling rate by suppressing one ormore samples that would otherwise have been sampled pursuant to thedefault rate.
 19. The mobile device of claim 12, further comprising atleast one of: the camera, a motion sensor, a light sensor, anorientation sensor, a microphone, a wireless communication interface, ora satellite receiver.
 20. The mobile device of claim 19, wherein theprocessing unit is configured to determine the state of the user byclustering sensor input from one or more sensors.
 21. The mobile deviceof claim 12, wherein the processing unit is further configured todetermine the state of the user based on a map of a building.
 22. Themobile device of claim 12, wherein the processing unit is furtherconfigured to determine the state of the user based on informationreceived from a calendar application executed by the mobile device. 23.The mobile device of claim 12, wherein the mobile device comprises ahead-mounted display.
 24. The mobile device of claim 12, wherein themobile device comprises a mobile phone.
 25. An apparatus comprising:means for causing a camera communicatively coupled with a mobile deviceto sample at a sampling rate; means for determining a state of a user ofthe mobile device based on sensor input; and means for modulating thesampling rate based on the determined state of the user.
 26. Theapparatus of claim 25, further comprising: means for selecting thecamera from a plurality of cameras, based on contextual information;wherein the means for causing the camera to sample at the sampling rateare configured to cause the camera to sample at the sampling rate basedon the selection of the camera.
 27. The apparatus of claim 26, whereinthe means for determining the state of the user comprises means fordetermining at least one of: a location or coordinate of the mobiledevice, a position of the mobile device in relation to the user of themobile device, an activity engaged in by the user of the mobile device,a place of relevance of the mobile device, or a change in surroundingsof the mobile device.
 28. The apparatus of claim 25, wherein the meansfor determining the state of the user comprise means for determining amotion state of the user.
 29. The apparatus of claim 28, wherein themeans for determining the motion state of the user comprise means fordetermining that the user is at least one of: stationary, walking,running, in transit, or fiddling with the mobile device.
 30. Theapparatus of claim 25, wherein the means for determining the state ofthe user comprise means for determining the user is involved in at leastone activity selected from the following list of activities: in ameeting, on a telephone call, being alone, watching TV, playing,reading, writing, meeting people, looking at screen, giving apresentation, eating, shopping, cooking, attending a talk, exercising,listening to music, or sleeping.
 31. The apparatus of claim 25, wherein:the sampling rate comprises a default rate; and the means for modulatingthe sampling rate comprise means for suppressing one or more samplesthat would otherwise have been sampled pursuant to the default rate. 32.The apparatus of claim 25, further comprising means for receiving thesensor input from at least one of: the camera, a motion sensor, a lightsensor, an orientation sensor, a microphone, a wireless communicationinterface, or a satellite receiver.
 33. The apparatus of claim 32,further comprising means for clustering sensor input from one or moresensors.
 34. The apparatus of claim 25, further comprising means fordetermining the state of the user based on a map of a building.
 35. Theapparatus of claim 25, further comprising means for determining thestate of the user based on information received from a calendarapplication executed by the mobile device.
 36. A non-transitorycomputer-readable medium having instructions embedded thereon foradaptively adjusting sensor sampling of an always-on camera, theinstructions including computer-executable code for: causing a cameracommunicatively coupled with a mobile device to sample at a samplingrate; determining a state of a user of the mobile device based on sensorinput; and modulating the sampling rate based on the determined state ofthe user.